Combined Gesture-Speech Recognition and Synthesis Using Neural Networks

Sign languages such as Spanish Sign Language (LSE) are the primary communication way among members of the Deaf community. However this language is not widely known outside of this community. The techniques for automatic recognizing hand signs proposed in this paper allow creating systems which can help deaf people to communicate with others, by providing them with computer tools for assisted communication or potentially with a fully automatic portable sign-language-to-speech translation system. The aim of this research it to implement a self-organizing neural network based technique for hand sign recognition, using self organizing map (SOM) and learning vector quantization (LVQ) based algorithms. The two classifiers are then combined to make the final decision. SOM and LVQ classifiers are training for the 26 signs of the manual alphabet and the speech synthesizer concatenate the different phones, diphones and triphones stored in the database to generate the right words in artificial speech. Finally, the system is to contribute to the implementation of meaningful human machine interactions in a workstation, mainly for welfare applications.